In order to ensure the security of audio signals in communication transmission, a fractional domain audio encryption algorithm based on cascaded chaotic system was proposed. Firstly, the audio signal was grouped. Secondly, the chaotic system was used to obtain the orders of fractional Fourier transform, and the order corresponding to each group data changed dynamically. Thirdly, the sampling fractional Fourier discrete transform with less computational complexity was used to obtain the fractional domain spectrum data of each group. Finally, the cascaded chaotic system was used to perform data encryption to the fractional domain of each group in turn, so as to realize the overall encryption of the audio signals. Experimental results show that the proposed algorithm is extremely sensitive to the key, and has the waveform and fractional domain spectrum of obtained encrypted signal more uniformly distributed and less correlated compared with those of the original signal. At the same time, compared with the frequency domain encryption and fixed-order fractional domain encryption methods, the proposed algorithm can effectively increase the key space while reducing the computational complexity. It can be seen that the proposed algorithm can satisfy the real-time and secure transmission requirements of audio signals effectively.